3.3 Tests and Their Environment

Sometimes, it doesn’t make sense to run a test due to missing preconditions. A required Emacs feature might not be compiled in, the function to be tested could call an external binary which might not be available on the test machine, you name it. In this case, the macro skip-unless could be used to skip the test:

(ert-deftest test-dbus ()
  "A test that checks D-BUS functionality."
  (skip-unless (featurep 'dbusbind))
  ...)

The outcome of running a test should not depend on the current state of the environment, and each test should leave its environment in the same state it found it in. In particular, a test should not depend on any Emacs customization variables or hooks, and if it has to make any changes to Emacs’s state or state external to Emacs (such as the file system), it should undo these changes before it returns, regardless of whether it passed or failed.

Tests should not depend on the environment because any such dependencies can make the test brittle or lead to failures that occur only under certain circumstances and are hard to reproduce. Of course, the code under test may have settings that affect its behavior. In that case, it is best to make the test let-bind all such setting variables to set up a specific configuration for the duration of the test. The test can also set up a number of different configurations and run the code under test with each.

Tests that have side effects on their environment should restore it to its original state because any side effects that persist after the test can disrupt the workflow of the programmer running the tests. If the code under test has side effects on Emacs’s current state, such as on the current buffer or window configuration, the test should create a temporary buffer for the code to manipulate (using with-temp-buffer), or save and restore the window configuration (using save-window-excursion), respectively. For aspects of the state that can not be preserved with such macros, cleanup should be performed with unwind-protect, to ensure that the cleanup occurs even if the test fails.

An exception to this are messages that the code under test prints with message and similar logging; tests should not bother restoring the *Message* buffer to its original state.

The above guidelines imply that tests should avoid calling highly customizable commands such as find-file, except, of course, if such commands are what they want to test. The exact behavior of find-file depends on many settings such as find-file-wildcards, enable-local-variables, and auto-mode-alist. It is difficult to write a meaningful test if its behavior can be affected by so many external factors. Also, find-file has side effects that are hard to predict and thus hard to undo: It may create a new buffer or reuse an existing buffer if one is already visiting the requested file; and it runs find-file-hook, which can have arbitrary side effects.

Instead, it is better to use lower-level mechanisms with simple and predictable semantics like with-temp-buffer, insert or insert-file-contents-literally, and to activate any desired mode by calling the corresponding function directly, after binding the hook variables to nil. This avoids the above problems.